home
***
CD-ROM
|
disk
|
FTP
|
other
***
search
/
Aminet 30
/
Aminet 30 (1999)(Schatztruhe)[!][Apr 1999].iso
/
Aminet
/
util
/
pack
/
xpk_Develop.lha
/
xpk_Develop
/
Autodocs
/
xpksub.doc
< prev
Wrap
Text File
|
1998-11-21
|
14KB
|
360 lines
TABLE OF CONTENTS
xpksub.library/XpksPackerInfo
xpksub.library/XpksPackChunk
xpksub.library/XpksPackFree
xpksub.library/XpksPackReset
xpksub.library/XpksUnpackChunk
xpksub.library/XpksUnpackFree
xpksub.library/--sublibs--
xpkmaster.library/--general-- xpkmaster.library/--general--
GENERAL COMMENT: There is a lot unimplemented stuff. When you do not
find a proper documentation, do not use it.
XPK is a standard packer system, where xpkmaster.library does the whole
managment stuff and xpkXXXX.library's do the packing and unpacking stuff.
The following should explain how such a library is build.
XPK does its packing with chunks. This means not the whole data is
crunched in one pass, but part for part. This allows to do progress
reports and even to decrunch large files when you have less memory.
So you always need to set the flags XPKIF_PK_CHUNK and XPKIF_UP_CHUNK.
There are ARCHIVE and STREAM flags as well. This is future stuff and not
used at the moment.
Each library has 101 (0-100) different modes, which can be selected by
user. You either can produce one XpkMode structure using these 101 modes
(or even ignore them) or you may divide it into different XpkMode
structures (you need to set XPKIF_MODES flag). Anyway the line of XpkMode
structures must start with mode 0 and end with mode 100. If you support
different modes, a higher mode means always better crunching factor.
The XpksPackerInfo functions tells xpkmaster.library all necessary
information about a sub library. It is called after the library is loaded.
The xpkmaster.library stores the crunched length, uncrunched length and
a checksum of crunched data for every chunk, so the sub library does not
need to do that.
The sub library function XpksPackChunk is called for the first chunk till
the last chunk. These chunks may have different size, but it is
guaranteed that the first one is always the biggest. If an error occurs
or the file is finished, XpksPackFree is called. Same happens for
decrunching with XpksUnpackChunk and XpskUnpackFree.
The crunched data may depend on previous crunched chunks (e.g. a table
stored in chunk 1). So if we start a new crunch, XpksPackReset is called
to end this current one. Using XpkPackReset xpkmaster can force the
sub library to produce independent chunks.
NOTE: It is allowed to do such stuff depending on previous chunks, but as
seeking is disabled by that, it's not recommended. You need to set
XPKIF_NOSEEK flag for such clients. Libraries supporting seek normally
have an empty XpksPackReset function.
When your library supports encryption, you need to set XPKIF_ENCRYPTION
flag, when your library always needs a password, set XPKIF_NEEDPASSWD as
well. When the library does not preserve original file (it is lossy like
JPEG), then set XPKIF_LOSSY flag.
To make XPK really save, you get a safety margin. This means for crunching
the destination buffer is at least XPK_MARGIN bytes longer than the input
file - For decrunching the destination buffer is at least XPK_MARGIN bytes
longer then the decrunched data.
All information is transfered through XpkSubParams structure. Most
important are xsp_InBuf, xsp_InLen, xsp_OutBuf, xsp_OutBufLen.
xsp_InLen is size of input file for packing and size of crunched file for
unpacking. xsp_OutBufLen is always the real buffer length. This includes
the XPK_MARGIN value. Never exceed the buffer borders. XPKERR_EXPANSION
error code exists for such cases.
The chunksize information in you library information helps
xpkmaster.library to find the used chunk size (It's not always the user
selected value). Minimum and maximum values depend on your algorithm. The
Default value should be something around 50KB or a value which brings fine
results with your packer.
See below for additional comments.
xpksub.library/XpksPackerInfo xpksub.library/XpksPackerInfo
NAME
XpksPackerInfo - Get information about a packer sub-library
SYNOPSIS
info = XpkPackerInfo()
D0
struct XpkInfo *XpkPackerInfo(void);
FUNCTION
Returns all information about a packer sub-library for use
by the master library.
RESULT
info - pointer to struct XpkPackerInfo. see xpk/xpksub.h
for the meaning of all the fields.
SEE ALSO
XpksPackChunk()
xpksub.library/XpksPackChunk xpksub.library/XpksPackChunk
NAME
XpksPackChunk - Pack a chunk of data
SYNOPSIS
err = XpksPackChunk( param )
D0 A0
LONG XpksPackChunk(struct XpkSubParams *);
FUNCTION
Packs one chunk of data from memory to memory. The size of the
chunk will not surpass the MaxPkInChunk field of the packer info
structure. Chunks are numbered in case a library writes various
dictionaries at special places.
INPUTS
struct XpkSubParams, where the fields mean:
InBuf Pointer to the data to be compressed
InLen The number of bytes to pack
OutBuf Pointer to the memory area to write packed data to
OutBufLen Size of above memory area
OutLen Will be set to compressed length by sublib
Number The number of this chunk
Mode The sub-mode to be used
Password The password to be used for encryption
Sub[4] Here the sub library stores its private data
RESULTS
err - global xpk error code
OutLen - Number of bytes written
SEE ALSO
XpksPackReset(), XpksPackFree()
xpksub.library/XpksPackFree xpksub.library/XpksPackFree
NAME
XpksPackFree - Free buffers associated with packing process
SYNOPSIS
XpksPackFree( param )
A0
void XpksPackFree(struct XpkSubParams *)
FUNCTION
Frees all buffers the sub-library has allocated privately
while doing some packing
INPUTS
param - The XpkSubParams used for packing
SEE ALSO
XpksPackChunk(), XpksPackReset()
xpksub.library/XpksPackReset xpksub.library/XpksPackReset
NAME
XpksPackReset - Clears all state information
SYNOPSIS
XpksPackReset( param )
A0
LONG XpksPackReset(struct XpkSubParams *)
FUNCTION
Clears all internal tables and any other state information of the
packer. This has the effect, to guarantee independent unpacking
of the chunk to come. The packers call this function automatically
before returning XPKERR_EXPANSION (data could not be compressed).
Note: packers have to store all of their state relevant to packing
in XpkSubParams to ensure reentrantness.
INPUTS
param - The XpkSubParams used for packing
SEE ALSO
XpksPackChunk(), XpksPackFree()
xpksub.library/XpksUnpackChunk xpksub.library/XpksUnpackChunk
NAME
XpksUnpackChunk - Uncompress a chunk of data
SYNOPSIS
err = XpksUnpackChunk( param )
A0
LONG XpksUnpackChunk(struct XpkSubParams *)
FUNCTION
Decompresses one chunk of data.
INPUTS
param - The XpkSubParams used for unpacking
xsp_InBuf Pointer to the data to be uncompressed
xsp_InLen The number of bytes to unpack
xsp_OutBuf Pointer to the memory area to write unpacked data to
xsp_OutBufLen Size of above memory area
xsp_OutLen Must contain the decompressed size of the data
xsp_Number The number of this chunk
xsp_Password The password to be used for decryption
xsp_Sub[4] Here the sub library stores its private data
RESULT
err - global xpk error code
SEE ALSO
XpksUnpackFree()
xpksub.library/XpksUnpackFree xpksub.library/XpksUnpackFree
NAME
XpksUnpackFree - Free all private data the sublib has allocated
SYNOPSIS
XpksUnpackFree( param )
A0
LONG XpksUnpackFree( struct XpkSubParams * )
FUNCTION
Will free all memory the sub-library has allocated during the
decompression.
INPUTS
param - The XpkSubParams used for unpacking
SEE ALSO
XpksUnpackChunk()
xpksub.library/--sublibs-- xpksub.library/--sublibs--
Some remarks for sublib writers
-------------------------------
First of all, Read the documentation. If you have got any further questions
writing xpksublibraries, either contact one of the XPK maintainers.
The name of a sub library is xpkXXXX.library, where XXXX is a combination
of uppercase characters ('A' to 'Z') and numbers ('0' - '9').
The first character must not be a 'X' (xpkXTRA.library is not allowed), as
this letter is reserved for xex libraries.
How XpkPackChunk works:
- Check xpar->xsp_Sub[0]. If it is 0, this is the first call. Allocate your
tables and store pointers to them in xsp_Sub[0] .. xsp_Sub[3].
- Read xpar->xsp_InLen bytes from xpar->xsp_InBuf and pack them to
xpar->xsp_OutBuf. Do not exceed xpar->xsp_OutBufLen. Store packed len in
xpar->xsp_OutLen.
How XpkUnpackChunk works:
- Check xpar->xsp_Sub[0]. If it is 0, this is the first call. Allocate your
tables and store pointers to them in xsp_Sub[0] .. xsp_Sub[3].
- Read xpar->xsp_InLen bytes from xpar->xsp_InBuf and unpack them to
xpar->xsp_OutBuf. Do noot exceed xpar->xsp_OutBufLen. Maybe compare
uncompressed len with xpar->xsp_OutLen.
Other notes:
- Your library should, of course, be re-entrant. This means no PC-relative
addressing of writeable data. Whatever you have, store a pointer to it
somewhere in the XpkSubMessage. In short: All state information of the
sublib is in the XpkSubMessage. You can detect the first chunk of a file
by the fact that this pointer is still NULL. In case your lib is not
re-entrant, return XPKERR_LIBINUSE when required.
- There will be no larger chunks on (de)compression than the first one.
(But with Seek you may not get first chunk really as first!)
- On compression, you may expand the data by XPK_MARGIN bytes maximum. This
means the output buffer is as large as the input buffer plus XPK_MARGIN.
Compression libs would, of course, already return XPKERR_EXPANSION if
they exceed the input buffer.
- You must supply a function XpkPackReset, which clears all tables so
that the next chunk will be able to be unpacked independently. You also
have to reinitialize all state information [calling your XpkPackReset]
before returning XPKERR_EXPANSION!
- On decompression, you will also have a XPK_MARGIN byte saftey margin for
runaway unpacking.
- Fill in one XpkInfo structure. See xpk/xpksub.h for the meanings of
the fields therein.
- If your packer has only one packing mode, fill in one XpkMode structure
(see xpk.h). Set the 'xm_Upto' field to 100, this means this struct
handles all modes up to (and including) 100, which is the max packing
mode. Place a pointer to this struct in the XpkInfo structure.
- There must be at least one XpkMode structure, and the last XpkMode
structure must have the xm_Upto set to 100. It must!
- If your packer has several modes, create several XpkMode structures and
sort them by efficiency in ascending order. Place a pointer to them
in XpkInfo. Set the XPKIF_MODES flag in the xi_Flags field of XpkInfo.
- If your packer cannot decrunch the chunks in any order, set XPKIF_NOSEEK
flag.
- Your packer will get the 0..100 number and is responsible of mapping
it to its own packing modes.
- Set the xi_DefMode field to some 0...100 number.
- If XSF_STEPDOWN is set in the xsp_Flags field of the SubParams structure,
you are allowed to reduce packing efficiency in order to save mem.
- Encryption libraries should compute a cryptographic hash [that is a
checksum] while or after decryption and compare it with one stored in the
encrypted data. If the hash of the decrypted text does not match the
stored checksum, return XPKERR_WRONGPW. Note, that this hash inevitably
discloses information applicable to breaking the code. Try to keep this
information small, e.g. by using a 16bit hash. Take care, that one has to
decrypt the whole chunk before it is possible to compute the hash of the
decrypted data.
How to get values for xm_PackSpeed, xm_UnpackSpeed, xm_Ratio?
-------------------------------------------------------------
Older versions of xpkmaster.library defined an Amiga A3000/25 with SCRAM
and AmigaVision executable as standard test for pack algorithms.
I searched for better testing defines and defined a new system which should
be used now and in future:
At http://corpus.canterbury.ac.nz/ you find a big set of files called
"Canterbury Corpus". (Not the large one, but standard test archive). This
is a collection of standard test files which can be used to test the
packing algorithms.
Speed values are still a problem, as it is hard to get a value usable on
all machines. New values should be computed on an A4000 with 68060/50Mhz
with no startup-sequence bootup. This defines a big field of different
machines which differ in RAM size, bus speed and other parameters, but the
test must be usable and so I cannot define one special environment.
The values entered in XpkMode structure are a sum of all test files.
You need to pack all files and note their results (reached file size,
crunch time, decrunch time). Use xBench to do that, as this is very
accurate (as good as possible).
Now add input file size (INSIZE), crunched file size (CRSIZE), needed
crunch time (CRTIME) and needed uncrunch time (UNCRTIME) for all of the
standard test files!
xm_Ratio = 1000*(1-(CRSIZE)/(INSIZE))
xm_PackSpeed = INSIZE/1024/CRTIME
xm_UnpackSpeed = INSIZE/1024/UNCRTIME
When you do not own an A4000 with 68060 CPU at 50Mhz then try to adapt
speed values.